A Comparative Prototype Research Methodology

1. Methodology Evaluating ubiquitous systems is hard, and has attracted the attention of others in the research community [5]. These investigators, like others in CSCW [3][6], argue there is a basic mismatch between traditional evaluation techniques and the needs posed by ubiquitous systems. Namely, these systems are embedded in a variety of complex real world environments that cannot be easily modeled (as required by theoretical analyses), simulated, measured, or controlled (as required by laboratory experiments). These concerns are shared by Abowd, Mynatt and Rodden, who argue “deeper evaluation results cannot be obtained through controlled studies in traditional, contained, usability laboratory.” [2]. As a result, many investigators have abandoned traditional comparative evaluation techniques and opted instead for techniques adapted from the social sciences, such as anthropology. We wanted to perform a comparative evaluation similar to a laboratory experiment, but in such a way that we could observe the effects of our design decisions in relatively unconstrained, real world use. This led us to the following process: 1. Design with alternatives. Pick a design decision to vary and build prototypes (or configurations of an existing system) to embody multiple design alternatives. a. Make the system variants provide the same basic underlying functionality. b. Make the underlying system functionality as simple as possible but at the same time compelling and useful enough to produce real usage. 2. Vary deployment situations. Deploy the prototypes in varied situations to help answer the question: are observed effects linked to a single situation or are they more general? 3. Compare and contrast. Use qualitative or quantitative data collection and analysis techniques to compare and contrast the alternatives and situations. a. Try to determine whether observed effects vary depending on the design alternatives, the varied situations, or both. In addition, the prototypes (or configurations) need to be useful – they need to be used by real people to address real problems and fit in with existing practices. This should be accomplished through techniques such as informal observation of current work practices and problems, or by using more principled studies. This evaluative method gives us a framework for better understanding a ubiquitous computing system. It goes beyond designing for use and promotes designing for evaluation. The framework produces a matrix of observations and measurements across situations of use that help us to think about ubiquitous computing design.